Custom changes in PagerDuty workflow

#1

Greetings!

I am new to PagerDuty and I’m tasked to come up with subtle changes in how PagerDuty primarily functions!

Here’s my requirement-

Notify 2nd level individual when an incident ran for more than 60 mins (Yes, despite level-1 acknowledging the incident, we would like to notify Level-2 as soon as clock go past 60 mins since the incident trigger)

At present, Pagerduty doesn’t notifies Level-2 if the incident is acknowledged by the level-1.

Note : we would like to notify level-3 if incident remains open for 120 mins or more so on and so forth for subsequent levels!

Is this possible at all? any leads on it is highly appreciated, Thanks!

#2

This would be possible with an external solution you’d need to develop outside of PagerDuty.

One approach might be:

  • Regularly query PagerDuty’s Incidents API for active, open incidents for certain services, priorities, etc. and store in a database or similar
  • Make additional API queries to get those incident’s log entries for timestamps of key activities and actions taken over the incident’s lifecycle
  • Evaluate the trigger timestamps against your resolution targets or other expected actions that may or may not have been taken yet
  • If resolution time has exceeded, escalate the incident to another layer of an escalation policy, add additional responders, run a response play, etc. as appropriate.